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Background / Context: 

Description of prior research and its intellectual context. 

While randomized experimental designs are the gold standard in education research 
concerned with causal inference, non-experimental designs are ubiquitous. For researchers who 
work with non-experimental data and are no less concerned for causal inference, the major 
problem is potential omitted variable bias. Rather than abandon efforts to test causal hypotheses, 
methodologists now recommend a variety of approaches, including triangulation across methods 
with careful attention to the logical propositions that underlie causal theory (see Duncan, 
Magnuson, & Ludwig, 2004; McCartney, Bub, & Burchinal, 2006). In this presentation, we 
provide an overview of a multi-year effort to follow this advice for the evaluation of a large- 
scale, school-based intervention for which a randomized design has not yet been possible to 
implement, but for which there are rich, longitudinal data and many opportunities to exploit 
design features to ask: does the intervention promote the achievement of children in high- 
poverty, urban schools? Specifically, we will present empirical results across estimation 
strategies in which we: (1) adjust for observed covariates, including baseline student 
characteristics, using propensity score methods such as weighting and matching; (2) compare 
treatment and comparison group differences in within-person growth (slope and intercept shifts) 
using fixed-effects interrupted time-series models; and (3) estimate the effect of one year of the 
intervention during elementary school on academic achievement using a comparative regression 
discontinuity approach. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

City Connects is a student support intervention that has demonstrated positive outcomes 
for elementary and middle school students (Center for Optimized Student Support, 2014; Walsh 
et ah, 2014). It is designed to address the out-of-school needs that can impact success in school 
for low-income, urban students. Having observed positive outcomes on academic achievement 
for students attending City Connects schools via quasi-experimental methods (Walsh et ah, 
2014), we seek to assess whether the findings can be considered a result of the intervention. Thus 
far it has not been feasible to implement a randomized controlled trial design since the 
intervention has grown over time in an organic way, adding schools at the invitation of school 
districts and funders. Randomization at the child level also is not possible since serving all 
students within a school is a critical feature of City Connects. In the absence of randomized 
design, we have developed the best possible evidence for the effectiveness of City Connects 
through multiple approaches to addressing potential selection effects resulting from violations of 
random assignment. The overarching research question is, “What is the effect of City Connects 
on student academic achievement?” In this paper we present three studies that support the 
argument that observed results can be attributed to City Connects. 

Study 1: Cross-sectional multi-level regression models with propensity score adjustment. 

Study 2: An individual-level fixed effects model approach to estimating achievement 
score change for City Connects students relative to comparison students. 

Study 3: A comparative regression discontinuity approach estimating the effect of one 
year of City Connects participation during elementary school on student academic achievement 
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Additionally, we will share results of sensitivity analyses that assess the robustness of 
estimated treatment effects to the presence of hidden selection bias and will briefly discuss 
results from studies of multiple City Connects sites. 

Setting: 

Description of the research location. 

The setting for this study is the Boston Public School (BPS) district, a high poverty, 
urban school district. Data for the studies presented in this paper come from 17 treatment and 90 
comparison elementary and K-8 schools in Boston, the largest city in Massachusetts. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The samples for the three primary studies presented consist of students attending 
kindergarten through fifth grade in BPS during school years 1999-2000 through 2012-2013. 
Specific sample sizes for each study vary, but the total sample includes approximately 2,500 
students in intervention schools and 15,000 in comparison schools, per grade. Table 1 displays 
demographic characteristics of City Connects students and comparison students in 2012. In 
general, about 85% of students in City Connects and comparison schools are low-income 
(eligible for free or reduced-price lunch) and most (about 90%) are students of color. There are 
some significant differences between students in the intervention versus comparison schools: 

City Connects students are more likely to be Asian, speak a first language other than English, 
and have higher rates of mobility (changing schools) compared to comparison students. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

Developed in 1999, City Connects is an evidence-based approach to school-based student 
support that partners with community agencies and institutions to promote the strengths and meet 
the needs of students in high-poverty urban schools. The program began in one school in Boston, 
Massachusetts, and grew to include schools in three states. In 2014-15, City Connects will be 
implemented in 63 schools in Massachusetts, Ohio, and New York. A full-time City Connects 
staff member in each school is at the core of the intervention. During the Whole Class Review, 
each classroom teacher and the School Site Coordinator discuss each student and document his or 
her strengths and needs across four domains (academic, social/emotional/behavioral, health, and 
family). Based on these strengths and needs, the School Site Coordinator then connects students 
with a tailored set of prevention, intervention, and enrichment services. Students with more 
intensive needs receive an Individual Student Review as needed throughout the year. In a 
proprietary Web-based system, the Coordinator documents and tracks the service plan for each 
student. A documented, standardized set of practices, oversight mechanisms, and fidelity tools 
guide implementation across sites and services. In addition to tracking data on students’ strengths, 
needs, and service referrals, our team has also been given access to a variety of student outcomes 
such as standardized test scores, report card scores for academic and behavioral measures, and 
demographic data. 

Research Design: 

Description of the research design. 

The City Connects evaluation employs a longitudinal quasi-experimental design with the 
intervention at the school level (all students within a given school receive the intervention). All 
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non-City Connects schools in the same district serve as comparison. In this paper we share 
results from three analyses that provide different pieces of evidence for a causal relationship of 
City Connects treatment to student outcomes. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

For all three studies, student outcomes data was made available through administrative 
records from BPS. At the end of each school year, information on student outcomes, enrollment, 
and demographic characteristics was provided by the school district with anonymous student 
identifiers. The analytic methods used for each of the studies are detailed below: 

Study 1: Two-level weighted linear regression models were built to predict standardized 
MCAS (the standardized exam used in Massachusetts public schools) scores in English 
Language Arts and Math in Grades 3 to 8, report card grades in Reading, Math, Writing, 
Behavior, Work Habits, and Effort in Grades 3 to 5, and weighted GPA in Grades 6 to 8. 
Propensity scores weights were applied at the student level to make the treatment and control 
groups statistically equivalent at the baseline. The probability of students being in the 
intervention group was estimated through binary logistic regression models that included an 
extensive list of covariates. A series of dummy variables indicating City Connects dosage were 
placed at the student level (students with maximum years of dosage served as the reference 
group); and the City Connects treatment effect was estimated at the school level. Thus the 
regression coefficient associated with the treatment indicator at the school level showed the 
average difference in achievement between comparison students and City Connects students who 
received the maximum years of City Connects in elementary schools. 

Study 2: This multi-level interrupted time series design with intervention and comparison 
students is currently being modeled in HEM (Raudenbush & Bryk, 2002). Student level fixed- 
effects will also be included in the analyses. The treatment group is defined as students who were 
in 3’^‘^ grade when their school first implemented City Connects. The comparison group includes 
3 grade students who attended a non-City Connects school during the same years. Outcome 
variables are teacher-rated Reading, Math, Effort, Work Habits and Behavior report card scores 
in grades 1 through 5. Eixed-effects analyses provide excellent controls for possible selection 
effects, since each student serves as their own control. In this approach, time-invariant selection 
effects drop out of the model, yielding unbiased estimates. (McCartney, Bub, & Burchinal, 

2006). A potential threat to validity with interrupted time series designs is history (any events 
occurring between the start of the treatment and the posttest that might have contributed to the 
outcome; Shadish, Cook & Campbell, 2001). By including both treatment and control groups 
from the same cohort of students, we help to reduce the possibility that a historical event other 
than treatment influenced the outcome. 

Study 3: We used a comparative regression discontinuity design in which birth date 
captured discontinuity-based effects for City Connects schools (e.g., students who were bom 
after the cut-off date for Kindergarten entrance were allocated to a later school year). In a 
comparative approach, we used the same definition to identify comparisons in non-City 
Connects schools. Regression -based continuity effects were compared across treatment 
conditions. The outcome variables used were grades 3 to 5 English/Language Arts and Math 
MCAS scores and proficiency levels. A comparative regression discontinuity design is based on 
known selection criteria, which can be modeled. This method normally assumes that if there 
were no intervention at the cut-off point, there would be a smooth transition at the discontinuity, 
but this comparative approach tests that assumption. 
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Findings / Results: 

Description of the main findings with specific details. 

Taken together, the results of these three studies suggest that positive outcomes for 
students who receive the City Connects intervention are evident when a number of different 
methodologies aimed at reducing omitted variables bias are used. 

Study 1: For elementary school outcomes, City Connects had positive impacts on some of 
the report card measures in Grade 4 and on most report card scores in Grade 5. City Connects did 
not demonstrate any impact on MCAS measures in elementary school. However, for middle 
school outcomes. City Connects had strong and positive impacts on both MCAS ELA and Math 
as well as GPA in almost every grade 

Study 2: Analyses are still in progress for this research question, but we expect that 
students most at risk (Tier 3) will experience the greatest growth in achievement after the 
introduction of City Connects at Grade 3. Preliminary models support this hypothesis. 

Study 3: Positive City Connects effects were found in both 3'^‘^ and 5* grade math and 
reading scores. Cohen’s effect sizes for analyses using MCAS raw scores as outcomes ranged 
from approximately .20 to .28. Increases in the likelihood of achievement at the proficient level 
or higher ranged from approximately 37% to 43%. Figure 1 shows the results of the comparative 
regression discontinuity using 5* grade MCAS math proficiency-plus status (scores of proficient 
and higher) as the dichotomous outcome. For the school year prior to City Connects 
implementation. City Connects and comparison students show statistically indistinguishable 
relationships between age and proficiency. For the year post-City Connects implementation, for 
both groups the form of the relationship between age and proficiency is similar to the previous 
year’s form, however, the intervention intercept is shifted upwards, indicating an overall positive 
shift in the proficiency levels of students in City Connects Schools. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

We have taken a logic -building approach to understanding the City Connects program. 
Several of our findings are consistent with what we would expect to see if the outcomes are a 
result of the intervention. For example, we see dosage effects: more time in the program is 
associated with more improvement in outcomes. We see evidence that we have successfully 
addressed selection bias: after applying propensity weights to take pre-intervention differences 
into account. City Connects and comparison students are comparable. We see results replicate: 
the results of our evaluative review demonstrate the positive effects of City Connects repeatedly. 
Given all of these considerations, we are increasingly convinced that the results are likely a result 
of the intervention. 

Despite the strengths of the methodology used in these studies, there are also limitations. 
For example, the research approaches presented here help to remove overt bias in observational 
studies; however, hidden bias from unobserved characteristics may still remain. Report card 
measures used in the longitudinal analysis may be limited in regard to measurement invariance 
since they are teacher-related. However, in Boston, elementary school report card scores are 
assigned in a standard way across the city, with a large component of each grade based on 
student performance on common subject tests administered to all students. We recommend that 
other school-based interventions that cannot feasibly use a randomized control design instead 
employ a range of methods aimed at reducing endogeneity and getting closer to understanding 
whether a causal relationship might exist between the intervention and its outcomes. 
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Table 1. Demographic Characteristics of City Connects Students, 2011-12 



Comparison 

Schools 

City Connects 

% Female 

48.3 

47.9 

Race/Ethnicitv 



% Black 

33.2 

31.5 

% White 

13.8 

7.2 

% Asian 

5.6 

10.8 

% Hispanic 

44.8 

48.0 

% Multi-Race Non Hispanic/Other 

2.6 

2.5 

% First Language Not English 

40.7 

49.1 

% English Language Learners 

14.9 

18.1 

Poverty: Eligible for Lunch Subsidy 



% Reduced Price Lunch 

4.6 

3.8 

% Lree Lunch 

67.8 

76.3 

% Special Education 

17.7 

21.0 

Mobility (% Attending Same School) 

78.3 

69.4 

Average Number of School Absences 

6.8 

7.4 


Figure 1. Probability of Achieving MCAS Mathematics Proficiency/ Advanced Categories for 5* 
Grade City Connects (Treatment) and Comparison Students 
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